The effect of examinee and patient ethnicity in clinical-skills assessment with standardized patients.

CONTEXT: Ethnicity has been a continuing concern for the valid assessment of clinical performance with standardized patients (SPs). The concern is that examinee ethnicity and SP ethnicity might interact, such that examinees might score higher in encounters with SPs of the same ethnicity. OBJECTIVE: To test for an interaction of examinee ethnicity and SP ethnicity on clinical performance in an SP examination. MAIN OUTCOME MEASURES: History-taking and physical-examination scores and interpersonal-and communication-skills scores, both based on checklists completed by SPs. Poststation scores for answers to case-related questions concerning pathophysiology, diagnosis, test selection, and test interpretation. SETTING AND PARTICIPANTS: Two graduating classes of over 1,000 fourth-year medical students each in the New York City Consortium were tested on the SP assessment administered at The Morchand Center of Mount Sinai School of Medicine. DESIGN AND ANALYSIS: The primary analyses were two-way (2 x 2) analyses, to test the main and interaction effects of examinee ethnicity and SP ethnicity. Effect-size measures (standardized mean differences, d) were computed to provide a sharper picture of the effects. RESULTS: Of the 24 interaction analyses, only three were statistically significant (not significantly more than expected by chance) and the results were mixed: one analysis showed better examinee performance in encounters with SPs of the same ethnic background and the other two showed the opposite. For all 24 interactions, significant or not, the results showed weak effects and no consistent pattern. White examinees scored on average 0.12 standard deviations above black examinees in encounters with white SPs, and 0.11 standard deviations higher in encounters with black SPs. CONCLUSIONS: These initial results are encouraging and should dispel some of the concern about ethnicity in SP assessment, at least about the operation of an examinee-by-SP-ethnicity interaction that would pose a serious threat to the validity of the examination scores.

Error patterns of 3rd-year medical students on the cardiovascular physical examination.

BACKGROUND: Recent research documents widespread deficits in the physical examination skills of practicing physicians. PURPOSE: This study explored physical examination skills of 3rd-year medical students after completion of a course in physical diagnosis. METHODS: Standardized patient physical examination checklist data were analyzed for a cohort of 2,038 medical students for a patient presenting with classic signs and symptoms of an acute myocardial infarction. A follow-up paper case and survey explored reasons underlying omissions. RESULTS: Students systematically omitted 3 of 10 component maneuvers critical to the evaluation of a patient with shortness of breath and chest pain. The same pattern of omissions was observed across 8 medical schools and over 2 successive years. The paper case follow-up study ruled out time constraints and performance anxiety as the cause. Survey data revealed that students may omit a maneuver due to inability to recall pertinence (blood pressure in both arms) or difficulty discriminating findings (heart sounds at different locations), or because of inadequate technical mastery (percussion of the lungs). CONCLUSIONS: These data suggest fundamental inadequacies in the current paradigm for teaching physical examination skills. Standardized patient checklist data can provide an informative window into the efficacy of teaching practices.

Effectiveness of problem-based learning curricula: research and theory.

PURPOSE: This article provides a critical overview of problem-based learning (PBL), its effectiveness for knowledge acquisition and clinical performance, and the underlying educational theory. The focus of the paper is on (1) the credibility of claims (both empirical and theoretical) about the ties between PBL and educational outcomes and (2) the magnitude of the effects. METHOD: The author reviewed the medical education literature, starting with three reviews published in 1993 and moving on to research published from 1992 through 1998 in the primary sources for research in medical education. For each study the author wrote a summary, which included study design, outcome measures, effect sizes, and any other information relevant to the research conclusion. RESULTS AND CONCLUSION: The review of the literature revealed no convincing evidence that PBL improves knowledge base and clinical performance, at least not of the magnitude that would be expected given the resources required for a PBL curriculum. The results were considered in light of the educational theory that underlies PBL and its basic research. The author concludes that the ties between educational theory and research (both basic and applied) are loose at best.

Global ratings of videotaped performance versus global ratings of actions recorded on checklists: a criterion for performance assessment with standardized patients.

PURPOSE: To test whether global ratings of checklists are a viable alternative to global ratings of actual clinical performance for use as a criterion for standardized-patient (SP) assessment. METHOD: Five faculty physicians independently observed and rated videotaped performances of 44 medical students on the seven SP cases that comprise the fourth-year assessment administered at The Morchand Center of Mount Sinai School of Medicine to students in the eight member schools in the New York City Consortium. A year later, the same panel of raters reviewed and rated checklists for the same 44 students on five of the same SP cases. RESULTS: The mean global ratings of clinical competence were higher with videotapes than checklists, whereas the mean global ratings of interpersonal and communication skills were lower with videotapes. The correlations for global ratings of clinical competence showed only moderate agreement between the videotape and checklist ratings; and for interpersonal and communication skills, the correlations were somewhat weaker. CONCLUSION: The results raise serious questions about the viability of global ratings of checklists as an alternative to ratings of observed clinical performance as a criterion for SP assessment.

Relationship between clinical competence and interpersonal and communication skills in standardized-patient assessment.

PURPOSE: To evaluate the relationship between clinical competence and interpersonal and communication skills, in an attempt to clarify current thinking about these two dimensions as measured with standardized-patient (SP) examinations. METHOD: Simple Pearson correlations were computed between total examination scores for clinical competence and interpersonal and communication skills. Three sets of different types of data involving 15 separate examinations were used to explore the generality of the findings. To control for a within-case halo effect and measurement error, corrected cross-half correlations and corrected cross-case correlations were also computed. RESULTS: The simple correlations and the corrected cross-half and cross-case correlations showed moderate and above relationships between these two dimensions in the clinical context. The simple correlations centered around .50, and the corrected cross-half and cross-case correlations were slightly higher, centering around .65 and .70, respectively. CONCLUSION: The authors' thinking is that the moderate relationship between clinical competence and interpersonal and communication skills is not due to a flaw in the measurement of clinical competence, as has been suggested, but rather is a natural consequence of the clinical encounter, which exacts an interdependence of these two dimensions. At least, this possibility must be seriously considered so medical educators can think and act appropriately in the assessment of clinical performance.

Effects of metrifonate on cognitive decline in Alzheimer disease: a double-blind, placebo-controlled, 6-month study.

Forty-seven patients with probable Alzheimer disease (AD) completed a 6-month double-blind study to compare metrifonate with placebo. The Alzheimer Disease Assessment Scale cognitive subscale score of the metrifonate group treated to a 50-70% inhibition of red blood cell acetylcholinesterase activity differed significantly from the placebo group score by 1.8 points (p < 0.03) due to a deterioration in cognitive performance in the placebo group (p < 0.01). Statistically significant deterioration also occurred in the Mini-Mental State Examination scores (p < 0.01) in the placebo-treated group. Adverse effects were uncommon and did not require adjustment of the dose of metrifonate or discontinuation of treatment. These findings extend our previous report of a favorable effect of metrifonate on cognitive symptoms in AD by showing clinical, not only statistical, significance.

Validating the standardized-patient assessment administered to medical students in the New York City Consortium.

PURPOSE: To test the criterion validity of existing standardized-patient (SP)-examination scores using global ratings by a panel of faculty-physician observers as the gold-standard criterion; to determine whether such ratings can provide a reliable gold-standard criterion to be used for validity-related research; and to encourage the use of these gold-standard ratings for validation research and examination development, including scoring and standard setting, and for enhancing understanding of the clinical competence construct. METHOD: Five faculty physicians independently observed and rated videotaped performances of 44 students from one medical school on the seven SP cases that make up the fourth-year assessment administered at The Morchand Center of Mount Sinai School of Medicine to students in the eight member schools in the new York City Consortium. RESULTS: The validity coefficients showed correlations between scores on the examination and the overall ratings ranging from .60 to .70. The reliability coefficients for ratings of overall examination performance reached the commonly recommended .80 level and were very close at the case level, with interrater reliabilities generally in the .70 to .80 range. CONCLUSION: The results are encouraging, with validity coefficients high enough to warrant optimism about the possibility of increasing them to the recommended .80 level, based on further studies to identify those measurable performance characteristics that most reflect the gold-standard ratings. The high interrater reliabilities indicate that faculty-physician ratings of performance on SP cases and examinations can or may be able to provide a reliable gold standard for validating and refining SP assessment.

The effect of formal feedback sessions on test security for a clinical practice examination using standardized patients.

OBJECTIVE: To determine the effects of formal feedback on test security for a clinical practice examination administered to successive rotations over a year's time. METHOD: A seven case clinical performance examination was administered to ten rotations of students throughout an academic year in a required Ambulatory Care Clerkship. Three of the cases were developed at Saint Louis University School of Medicine (SLU); four of the cases were developed by the National Board of Medical Examiners (NBME). Immediately following each examination, the examinees participated in a feedback session for just the three SLU cases: checklists were displayed, and details of the cases were discussed. No feedback on the four NBME cases was provided. Student performance in successive rotations on the three cases in which students were given detailed feedback by the faculty was compared with performance on the four cases in which no feedback was given. SAMPLE: Data were collected from 119 students. RESULTS: The interaction of case and group was not significant, indicating no difference in the trend over the ten groups between the three cases for which feedback was provided and the four cases for which it was not. The results showed no significant increasing trend over the ten rotation groups for either cases where feedback was given or withheld. CONCLUSION: The results suggest that the formal feedback sessions did not pose a threat to test security.

Using standardized patients for assessing clinical performance: an overview.

Standardized patients are being used increasingly worldwide to assess the clinical skills and competencies of medical students, residents, and physicians; plans are now underway for the use of standardized patients in licensure and certification examinations. The Morchand Center for Clinical Competence was established at Mount Sinai School of Medicine for the purpose of assessing clinical performance with the use of standardized patients, and currently all students in the eight medical schools in the New York City Consortium are tested at The Morchand Center. This paper presents an overview of standardized-patient assessment, a description of the assessment program at The Morchand Center, and a summary of research on this assessment approach including research at The Morchand Center.

Ventures in education: a pipeline to medical education for minority and economically disadvantaged students.

BACKGROUND: Ventures in Education is an independent, nonprofit educational organization established by the Josiah H. Macy, Jr. Foundation to improve the academic achievement of minority and economically disadvantaged students, particularly in science and mathematics. One specific objective has been to increase the number of students who enter schools of the health professions, in particular schools of medicine, which was the focus of this study. METHOD: A search was conducted of the Association of American Medical Colleges' (AAMC's) Student and Applicant Information Management System database, to determine whether any of the 981 graduates in the first five Venture classes (1985 to 1989) of the original five New York City high schools in the program had pursued medical education. RESULTS: The search located 160 of the 981 Ventures graduates, and, of those, 136 had taken the Medical College Admission Test (13.9%), 109 (11.1%) has applied to medical school, 75 (7.6%) had been accepted, and 72 (7.3%) had matriculated into medical school. All of these percentages were significantly higher than the corresponding percentages for the general population. CONCLUSION: The findings have important implications for the AAMC's Project 3000 by 2000, showing that a rigorous academic curriculum with resources for individualized attention can facilitate the entry of minority and economically disadvantage students into medical education, with at least 7.3% of the Ventures graduates entering medical school and nearly 70% of those applying subsequently being accepted.

Psychometric properties of a standardized-patient checklist and rating-scale form used to assess interpersonal and communication skills.

The results show that the SP checklist scores and the SP ratings of interpersonal and communication skills have comparable psychometric properties. The reliabilities of the five-item rating form (.76) and the single global rating of patient satisfaction (.70) were slightly higher than the reliability of the 17-item checklist (.65); this finding is of particular significance, given the greater length of the checklist. Also, the checklist scores and ratings appear to be measuring the same underlying dimension, with correlations of the checklist with the five ratings and with the single global rating being .82 and .81, respectively. Van der Vleuten and associates, in two excellent articles, noted a recent shift away from the use of subjective measures of clinical competence, such as rating scales, toward the use of presumably more objective measures, such as SP checklists. Their concern was that these objective measures may focus on somewhat trivial and easily measured aspects of the clinical encounter, and that more subtle but critical factors in clinical performance may be overlooked or ignored. They referred to such measurement as "objectified" rather than objective. The shift is based on the presumption that objective or objectified measurement is superior to subjective measurement, such as ratings, with respect to psychometric properties such as reliability. On the basis of a survey of several studies, though, the authors concluded that "objectified methods do not inherently provide more reliable scores" and "may even provide unwanted outcomes, such as negative effects on study behavior and triviality of the content being measured." The results of the present study support this conclusion, showing somewhat higher reliabilities for subjective ratings than for the objective (or perhaps objectified) checklist. Also, the high uncorrected correlations suggest that the more reliable ratings are measuring the same underlying dimension as are the checklist scores. The present study also illustrates the application of a recently proposed method for constructing a valid SP checklist, which would consist of items that best reflect global ratings of performance. In this study, the ratings were provided by the SPs themselves, but ratings could be obtained from faculty-physician experts who observe student performance on the SP case. Thus, performance on individual checklist items would be correlated with expert ratings, to identify the items that best predict the ratings. The checklist, then, would be constructed of just those items that best predict the ratings, and the checklist could be used for future testing without the need for further faculty ratings (yet the checklist scores would reflect the faculty ratings). With this approach, it would seem possible to construct checklists for history-taking and physical-examination skills, as well as for interpersonal and communication skills. Thus, the faculty ratings would provide a basis for case development and refinement, including scoring and standard setting, and scores on the checklist would serve as a proxy for the gold-standard faculty ratings. The study suggests that SP ratings may be more efficient and more reliable than SP checklists for assessing interpersonal and communication skills. The study also demonstrates that global ratings by SPs (or by expert physician observers) can provide a basis for SP-test construction.